{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Use Case 6 - Teleconnection explorer\n",
    "\n",
    "## Problem definition\n",
    "\n",
    "As part of a project on climatic teleconnection, a student investigates how El Niño-Southern Oscillation (ENSO) relates to monsoon rainfall. A result could be a plot showing the sliding correlation between Indian Summer Monsoon Rainfall (ISMR) and SST anomalies [RD-4]. A more sophisticated version of this task would be to calculate the Multivariate ENSO Index (MEI, [RD-5], [RD-6]). Additionally, also the comparison of the ENSO index with other CCI datasets (e.g. Cloud, Fire) would be interesting.\n",
    "\n",
    "## Required toolbox features\n",
    "\n",
    "- Access to and ingestion of ESA CCI SST and Soil Moisture data\n",
    "- Geometric adjustments\n",
    "- Spatial and temporal filtering and subsetting for both data sets\n",
    "- Calculation of anomalies and statistical quantities\n",
    "- Visual presentation of statistical results and time series\n",
    "- ENSO index calculation from SST data\n",
    "- Calculation of the absolute anomaly on the log transformed soil moisture data\n",
    "- Calculation of the correlation between the two time series with a lag of 30 days\n",
    "- Generation of a correlation map and export of the correlation data\n",
    "- Generation of a time series plot of the correlation by the selection of a location in South East Asia\n",
    "- Saving of the image and the underlying data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ingest data, spatial and temporal subsetting, variable selection\n",
    "\n",
    "To reduce the amount of data to download and later work with, it is beneficial to do spatial and temporal subsetting, as well as variable selection as early in the workflow pipeline as possible. Cate supports these operations at download time, or when opening a dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from cate.core.ds import DATA_STORE_REGISTRY\n",
    "import cate.ops as ops\n",
    "from cate.util import ConsoleMonitor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "monitor = ConsoleMonitor()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data_store = DATA_STORE_REGISTRY.get_data_store('esa_cci_odp')\n",
    "local_store = DATA_STORE_REGISTRY.get_data_store('local')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "local_store"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set regions of interest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "s_india = '72, 8, 85, 17' # 'lon_min, lat_min, lon_max, lat_max'\n",
    "pacific = '-175, -10, -115, 10' # 'lon_min, lat_min, lon_max, lat_max'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the desired spatial and temporal subsets and ECVs as separate datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "soil_sources = data_store.query('esacci.SOILMOISTURE.day.L3S.SSMV.multi-sensor.multi-platform.COMBINED.02-2.r1')\n",
    "soil_sources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Doesn't work, see Issue 256. The dataset is already downloaded\n",
    "#soil_sources[0].make_local(local_name='SOIL_2007',\n",
    "#                           region=s_india,\n",
    "#                           time_range='2007-01-01, 2007-12-31',\n",
    "#                           var_names='sm, sm_uncertainty',\n",
    "#                           monitor=monitor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sst_sources = data_store.query('esacci.SST.day.L4.SSTdepth.multi-sensor.multi-platform.OSTIA.1-1.r1')\n",
    "sst_sources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# The dataset is already downloaded\n",
    "#sst_sources[0].make_local(local_name='SST_2006_2007',\n",
    "#                          region=pacific,\n",
    "#                          time_range='2006-01-01, 2007-12-31',\n",
    "#                          var_names='analysed_sst, analysis_error',\n",
    "#                          monitor=monitor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Open and plot the first time slices of the downloaded datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "soil = ops.open_dataset('local.SOIL_2007')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "soil"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "ops.plot_map(soil, var='sm', time='2007-01-01', region=s_india, file='/home/ccitbx/Desktop/uc6_fig1.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sst = ops.open_dataset('local.SST_2006_2007')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "sst"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "ops.plot_map(sst, var='analysed_sst', time='2006-01-01', region=pacific, file='/home/ccitbx/Desktop/uc6_fig2.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Temporal aggregation, Long Term Averaging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Further operations (index calculation, long term averaging) work on monthly datasets. Hence, aggregate the daily datasets to monthly temporal resolution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sst_monthly = ops.temporal_aggregation(sst)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sst_monthly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "soil_monthly = ops.temporal_aggregation(soil)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "soil_monthly"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a long term averaged sea surface temperature dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sst_lta = ops.long_term_average(sst_monthly)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sst_lta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The index calculation operation reads the reference dataset from a given file name, hence, save the long term average dataset. This takes longer than previous operations, as most of the actual calculation is done in a streaming fashion, when writing the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ops.write_netcdf4(sst_lta, '/home/ccitbx/Desktop/sst_lta.nc')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ENSO index calculation and correlation analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate ENSO index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "enso_index = ops.enso_nino34(ds=sst_monthly,\n",
    "                             var='analysed_sst',\n",
    "                             file='/home/ccitbx/Desktop/sst_lta.nc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "enso_index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ENSO index result is in a tabular format, correlation takes a dataset, hence we convert the table to a dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "enso_ds = ops.from_dataframe(enso_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "enso_ds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to calculate correlation with a one month lag. Subset the datasets accordingly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "soil_monthly_jan_nov = ops.subset_temporal(soil_monthly, '2007-01-01, 2007-11-01')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "soil_monthly_jan_nov.time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "enso_index_dec_oct = ops.subset_temporal(enso_ds, '2006-12-01, 2007-10-01')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "enso_index_dec_oct.time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extract a point of Soil Moisture data to correlate with the ENSO index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "soil_monthly_jan_nov_point = ops.tseries_point(soil_monthly_jan_nov, (78, 12), 'sm')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "soil_monthly_jan_nov_point"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perform correlation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "corr = ops.pearson_correlation_scalar(enso_index_dec_oct, soil_monthly_jan_nov_point, 'ENSO N3.4 Index', 'sm')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "corr"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3.0
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}